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TEST 



RESOLVER MANAGEMENT SYSTEM 



g^old of tb '- Tnveation 
The present invention relates to a system for 
processing answers to test questions. 

M„lM| ni Tnvention 

The scoring of test answer sheets involves complex 
problems. These test answer sheets, typicall. include a 
series of response positions such as, for example, 

.^.nales A person talcing a test 
"bubbles," ovals, or rectangles. A p , , 

darken in an appropriate oval with a 
would, for example, darKen n 

penci. « answe. a ^uX.ipIe cKo.ce <^estio„. T^s. ..s. 
^s..r s.e.« .a. also .nclu.e nan.«..«e„ ans«e« sue. as 
.3sa. or She. ans»e. ^es.ions. S.s.e„s sca„..n, a^ 
sccin, ..e .u..les o„ sucK ans«e. sheets a.e .no« .n .he 

Xncreased di«i=ultles a.e encountered, hovever, »hen 
such answer sheets either include other types o. answers 

o or- cannot be machine graded - 
such as handwritten answers, or cannot 

1. if the student has failed to include hxs or 
For example r it Tine . 

to machine score the test answer. 

The ,oals in scoring test answers that cannot 
machine scored include etticiency and consistency. These 
test answer sheets are typically scored by test resolvers 
either hy ..anually scorin, the physical test answer sheet 



or scoring an electronic representation of the test answer 
Sheet on a computer. Idealiy, the scores provided by the 
various test resolvers for a particular test creation 
should be consistent, since the scores are used in 
co„parin, performance of the students against one another, 
in addition, a test resoiver should ideally worlc 
efficiently so as to maintain consistently high scoring 
rates. The test resoiver should not have such a high 
scoring rate that the consistency or guality of scoring 
Significantly declines, likewise, the test resoiver should 
not have such a low scoring rate that the too few answer 
sheets are being scored. This manual scoring o£ test 
answer sheets, however, .aKes it difficult to monitor the 
consistency of scoring among the various test resolvers. 

in many situations, test resolvers actually travel 
to a particular location so that all test resolvers may 
simultaneously score test answer sheets. Requiring the 
test resolvers to travel to a given location xs 
inconvenient for the resolvers and expensive for those who 
administer the tests. Furthermore, tracking the 
performance of test resolvers against both their own 
performance and the performance of other resolvers can be 
very difficult with a manual scoring environment. 

The process of resolving test questions is 
currently done manually, and this presents problems. K 
resoiver is manually presented with the actual test answer 



10 



sheets for scoring. This process is relatively 
inefficient, since the resolvers must score the answer 

presented. ^Iso. manual scorin, systems do not have the 
capability to efficiently gather and categorise the test 
answers for subsequent analysis. Therefore, with a manual 
system it is very difficult to determine how teaching 
methods should be changed to decrease, for example, the 
number of incorrect answers. 

A need thus exists for a system that promotes and 

efficiency in scoring or resolving 
achieves consistency and efficiency in 

of tests. 

g»n,marv of invention 

The present test resolver management system 
facilitates consistent, accurate, and high quality scoring 
of test answers. A collaborative scoring sub-system allows 
two test resolvers to score the same test answer and 
.esolves potential discrepancies in the scores provided by 

20 the resolvers • 

The system also has the capability to use quality 
items to monitor a resolver-s scoring performance. The 
system transmits a quality-control test item, having a 
Known score, to a test resolver. Kfter receiving a score 
25 from the resolver, the system compares the known score wxth 
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the resolver 's score and can generate a quality profile for 

the resolver. 

The system also provides performance feedback to 

test resolvers by monitoring the scoring rates of the 

5 resolvers. If a resolver's scoring rate, in comparison to 

another scoring rate, contains a discrepancy, the system 

can report the discrepancy. The system can also recommend 

breaks to a resolver, or provide a resolver with 

diversionary activities, in response to a discrepancy in 

10 scoring rates. The system also has the capability to route 

test answers to resolvers based on classifications of 

resolvers and persons who generated the test answers. 

Monitoring and selecting resolvers based on classifications 

promotes fair and consistent scoring of test answers. 

Finally, the system provides for an on-line 

scoring guide to dynamically provide a test resolver with 

scoring rules for a test answer displayed to the resolver. 

Brief Description of the Drawings 
20 FIG. 1 is a diagram of a network that incorporates 

the present invention. 

FIG. 2 is a block diagram of a portion of the 

network shown in FIG. 1. 

FIG. 3 is a block diagram of the scanning 
25 configuration in the network of FIG. 1. 
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„G. 4 is a bloc, diagram ol «>e server in .he 
„e„or. o.^.xa. I. ^ ^^^^^ ^^^^^^^ 

" i- no. c.r. o. ..i.iPie i.e. scori.. 

, . no. car. o. ca«,or..ed <spec.aX> 

"^^^":::'s-.Oareano.c.ar.o.ooUa.ora.ive 

.0. U is a no. car. o. <.aU.. 
'""^'"l;.. n is a no. Char, of reso.ver «o„i.orin, and 
"^"""■„0. „ is a no. Char. o. an on-une scorin, 

15 ^y^"^^"*- , of a user interface for use 

FIG. 14 is an example of a user 

with multiple item scoring. 

* 4.t,o PivTrrr^^ Embodiment 
DetailedJescriEtion.21^ — " 

. rf^tailed description of the 

in the following detaiiea a *r 
20 . to the accompanying 

— e-.d.en.. J^^n .hich is sho.n 

— "'^^'-r/asXi^^— n.in.hich.he 

Of iUus.ra..on spe ^^^^^^^^^ 

,„ven.ion .ay be pracced. .hrs 
„ . .nicien. de.au .o enable .os s^r .^^^^^^ ^^^^ 
practice the invention, and it xs 
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other embodiments may be utilized and that structural or 
logical changes may be made without departing from the 
scope of the present invention. The following detailed 
description is/ therefore, not to be taken in a limiting 
sense, and the scope of the present invention is defined by 
the appended claims. 

HARDWARE CONFIGURATION 
FIG. 1 illustrates an example of a hardware 
configuration for a network that incorporates the present 
invention. This configuration is shown as an example only; 
many different hardware configurations are available, as 
recognized by one skilled in the art, for implementing the 
software processing functions described below. The network 
shown comprises a mainframe computer 20 interfaced through 
a backbone token ring to a plurality of RISC servers 11, 12 
and 13. Each RISC server is interfaced to a token ring 
that contains work stations and scanners. The RISC server 
11 is connected in token ring 1 to scanners 14 and work 
stations 19. The RISC server 12 is connected in token ring 
2 to scanner 15 and work stations 18. The RISC server 13 
is connected in token ring 3 to scanners 16 and work 
stations 17. The mainframe computer 20 is also connected 
to a high capacity printer 22 and a low capacity printer 21 
25 for printing reports of stored data within the system. 
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The system use. the scanners for reading in test 
ans-er sheets. These test answer sheets .ay ==«prise. .or 
e««ple, test for^s with "bubbles" or ovals representing 
possible answers, handwritten essays, or other various 
.ypes of written or printed information. After receiving 
the scanned test data, the system within the RISC servers 
can process those scanned test answer sheets to generate 
.est items of interest from the answer sheets. . test item 
is, therefore, an electronic representation of at least a 
portion of a test answer sheet. The system may distribute 
these test items to the wor. stations for on-line scoring. 
^ test scorer at a work station can then score the test 

score The system receives the test 
item and enter a test score. i 

scores via the natwor. and the RISC servers and distributes 
.he scores to an appropriate computer for subsequent 
printing and reporting, the appropriate computer may 
include, for example, the mainframe computer 20 or a 
server. The system may also transmit the test scores to, 
for example, a disk or telephone line. 

FIG. 2 is a more detailed block diagram of a 
portion of the network shown in riG. 1. As shown in FIG. 
2, the scanning units shown in FIG. 1 typically comprise a 
scanner 25 interfaced to a computer 24 and personal 
computer ,PC, 26. FIG. 3 shows a more detailed block 

^4- ThP scanner 25 contains a 
diagram of a scanning unit. The scann 

^ f^^st answer sheet, and 

camera 31 for optically readxng xn a test 



further contains optical mark recognition (OMR) logic 32 
for processing the scanned data received from camera 31. 
The PC 26, preferably implemented with a high performance 
486-level PC, contains a frame buffer 23 for receiving the 
scanned image data from the scanner 25. 

The computer 24, preferably implemented with an HP 
1000, is interfaced to the scanner 25 and PC 26 for 
controlling the operation of the scanning unit. The 
computer 24 is optional; the system .ay alternatively be 
configured such that all of the functionality of the 
computer 24 is within the PC 26. The computer 24 controls 
the scanner via the OMR logic 32 and thus controls when 
i„.age data is scanned in and subsequently transferred to 
the PC 26. The PC 26 essentially acts as a buffer for 
holding the image data. The computer 24 further controls 
when the PC 26 will interrogate the image data for 
transmission to a server 27 for subsequent processing and 
scoring. The PC 26 can also electronically remove or 
«clip" an area of interest from the image data, which 
represents at least a portion of the scanned test answer 
sheets . 

Examples of two systems for storing and extracting 
information from scanned images of test answer sheets are 
shown in U.S. patent Nos. 5,134,669 and 5,103,490, both of 
which are assigned to National Computer Systems, Inc. and 
are incorporated herein by reference as if fully set forth. 
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0-7 receives the image data, which 
The seirver 27 receiv 

;,nd provides for processing and 
— = — - ^^^^ ^„,„„. .... - a 

co„«ol of .he ..age da.a. ^^^^^^ 

•i^ then distributed to the wor^ 
test item, xs then ^^^^^^ ^^^^^^^^ 

and 30 for subsequent scorxng. 

V station typically receives the test xtem, 
the work statxon typ receiving 

.„„ and transmits the score to the r 
the scorxng, ana ttai 

computer. hardware and 

PIG- 4 is a block diagram of the ha_^ 

. ■ in a server in the network of FIG. 1. A 
.nftware functions in a ser»= ,e 
software interfaces with the scanner PC 26 

control .oduxe 3 .nte ^^^^ ^^^^^ ^ 

receives the .... ^^^^^^^^^^^ 

7r t:;r;:tores — .f initions an. hanaun, 
(CAR) 33 typ functions as a 

The document process 
criteria. The a server 27. 

.uffer into a .ain processing module 45 m serv 

■ nrocessin, -odule 45 controls the 
..e .a.n proces g „.„3.issicn of 

test items to the work stations lor \ 

. ■ n of scores to the mainframe computer 20. 
, transmission of scor performance of 

„„rf„le 45 also monitors tne p<= 
„.in processing module ^^,^„t and efficient 

t.e test resolvers to main.in c^ 

.solving Of test ^'^^^ _ins 
The main process iny 
„ no basic functions, which are controlled by 
,5 the following basiC no. module 38 

system management module 32. 
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receives image data from the database 36 and controls the 
flow of data into an edit process module 39. The edit 
process module 39 may perform machine scoring of the test 
items. For those test items which cannot be machine 
scored, or possibly for other test items, the system 
t:ransmits such test items to the job build function 40. 
The job build function 40 determines what type of 
subsequent scoring is required for the test item and, for 
example, which work station will receive the test item. A 
job send module 41 receives the test item and transmits it 
to a router 42, which in turn transmits the test item to a 
send/receive communication module 43. Edit work module 34 
and edit server module 35 control the flow of test items 
into and out of server 27. Incoming data, such as test 
answers from the work station, are transmitted through 
„,odules 34 and 35 to a job receive module 44. The job 
receive module transmits the data to the edit process 
module 39 for subsequent storage within the database 36. 
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SOFTWARE PROCESSING 
FIG. 5 is a flow chart of typical scanning and 
processing of test and answer sheets. The document 
processing system receives the test answer sheets, or other 
documents, at step 50 and performs initial clerical 
preparation of the documents (step 51) for scanning at step 
52. The system at step 52 scans the documents for OMR and 
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. „ The syste. »ay then process the om 
.^.les at step 53 an. sto« 

,^ess sto.a,e ^/^^ ..,e. .he 

,.ep of clxpp 9 ^^^^ ^^^^ 

.yp^caUy .n ; comprise any 

._e. - _ e.»p.e. a han.«..tten 

eUl response positions. The system »ay aUo 
"irl --.y .o. .o.e.n souses, .a^et. 
.ece.ve «a,e . ^^^^ ^^^^ ,,,^„e 3S. 

Sn^se^ent operations on th ^^^^^ 

,3 to the source of the data. ^^^^^^ , est items 

. i-he system stores ^11^= 

interest from the .mage, the sy 

„ it., to .uXtipie ite. 

Otherwise, the syste. performs other 

.„ the .ata at step .0 an. stores - re^^^^^^^_^ 

process storage 55 at step • 

T« machine scoring, raw K«jr 
„ay include, for example, machxn 

25 and analytic resolving. 



11 



;^alytic resolving or scoring .ay include, for 

.omoarisons such as bit-mapped comparisons 
example, map comparisons 

. •i-^.ms The map comparisons allow a test 

between two test items. Tne map 

for example, the answers of a 
resolver to compare, for examp 

v« track the respondent's progress, 
respondent over time to track tn P 

..nd-drawn circles by .K. respondent .o determine .he 

.i„e. ^.Xytic scoring a.so Include, for ex^ple 

cxrcxin, or eXectronXcaUy indXcaUn, .XsspeXXed words and 

a« answer such as an essay, 
punctuation errors in an answer 

MultiEle_Ii:^m Scoring 
„G 6 is a £lo» chart o£ typical «ltiple ite» 
p.ocessin,. .he syste. at step typically Urst .etches 
. multiple its. i.a,e .ro„ the .or.-in-process stora,e^ 
..e i.a,e is stored in a multiple ite. display ^e^^ 
and a multiple ite» display storage « £or subse^ent 
display to a resolver. The syste. continues to recer.e 
multiple iteo,s until either the ite» display rs full, 
determined at step 6S, or no .ore multiple ite.« are 
p.esent as determined at step «. .s Ion. as the drsplay 
is not full and additional multiple items are present, 
system preferably scans the .or,-in-process storage at step 
, for additional items. When the multiple item display 

^.U or no more multiple items are present, the system 
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sends the compiled multiple items to a resolver at step 70 
and displays the multiple test items on. the resolver 
display 71. 

The system typically transmits test items to a 
particular resolver based upon the resolver-s resolution 
expertise. For example, a certain resolver may be assigned 
to score all of the test items relating to science 
questions. Resolution expertise may also comprise, for 
example, math, english. history, geography, foreign 
languages, or other subjects. 

An example of an interface on the resolver display 
IS Shown in FIG. 14. .he interface typically comprises a 
plurality of cells 74, with each cell containing one test 
item to be resolved. After displaying the ^Itiple items 
in the cells of the resolver display, the system allows the 
resolver at step 72 to score the multiple items. A test 
.esolver would typically indicate the score of the answers 
by using a "mouse," light pen, touch screen, voice input, 
or some other type of cursor control or input device. 

in the example shown in FIG. 14, the correct 
answer is "four" and the incorrect answers are indicated by 
the shading. Alternatively, a resolver could indicate the 
correct answers. The advantage of the multiple item system 
arises from the simultaneous display of test items in the 
cells 74, which allows a test resolver to guickly score 

A achieve a faster response time in 

many test items and thus achieve a 
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comparison to the display and scoring of only a single test 
item at a time. Even the simultaneous, display of two items 
increases response tune. As the matrix of cells increases, 
the simultaneous display of test items achieves a 
5 significant increase in response time and resolver 

attention and focus. 

After scoring or resolving, the system receives 
the results at step 73 for subsequent storage in work-in- 
process storage 55. A test resolver typically transmits 
10 the results of resolving all displayed test items in the 
cells as a single unit for batch processing. 

rateaorizftd Item R enorting 
FIG. 7 is a typical flow chart of categorized 
15 (special) item reporting. Categorized item reporting 
allows the system to both group answers according to 
predefined categories and monitor processes used by the 
students or test-takers in arriving at a given answer. The 
categories in which test answers may be grouped include, 
20 for example, incorrect answers and correct answers within a 
curriculum unit within an instructional grouping and 
requested time frames; for example, all of the incorrect 
„,ath answers in a particular instructor's class during the 
previous school year. Other groupings are possible 
25 depending upon the needs of the test resolvers and 
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instructors who teach the material to which the test 
relates • 

in addition, the system may merge an image of a 
test item with the corresponding score. In order to 
facilitate teaching of material to which the test relates, 
the system typically merges a test item representing an 
incorrect answer with the corresponding score. By 
reporting the actual test item, an instructor may gain 
insight into a thought process used by the student in 
arriving at the incorrect answer. Therefore, by having 
some knowledge of why a student answered a test question 
incorrectly, an instructor can take measures to change or 
modify teaching strategies to correct the situation. 

The categorized item reporting normally comprises 
the following functions. The system at step 75 scans the 
work-in-process storage for items that are ready to be 
reported. If test items are ready for reporting, as 
determined at step 76, the system processes the data at 
step 77 for generating an appropriate report of the data. 
At step 78, the system scans the central application- 
repository for definitions of categorized (special) items. 
AS special items are available for reporting, as determined 
at step 79, the system retrieves the special items at step 
80 and can merge it at step 81 with other report 
information, such as the corresponding test items, as 
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explained above. The system then distributes a report at 
step 82, which can be a printed report, 

r.ol labors tlve Scoring 
FIGS. 8-11 are a flow chart of a typical 
collaborative scoring system. The collaborative scoring 
system provides for functions to achieve fairness and 
Objectivity in resolving of test items. The collaborative 
scoring, for example, allows two resolvers to score the 
same item and, if the answers are not within a certain 
predefined range, provides for subsequent processing to 

resolve the discrepancy. 

The system at steps 83 and 84 determines if items 
are available for scoring. At step 85, the system receives 
collaborative scoring requirements from the database and 
determines at step 86 if collaborative scoring is required. 
Examples of collaborative scoring requirements are 
illustrated below. If collaborative scoring has been 
specified, the system retrieves the item to be scored from 
the work-in-process database at step 87 and sends the item 
to resolvers 1 and 2 at steps 88 and 91. 

The system is further able to choose resolvers 
according to selection criteria at steps 89 and 90. The 
selection criteria of the resolvers for scoring answers may 
include, for example, race, gender, or geographic location. 
The ability of the system to assign test resolvers to score 
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particular test itees provides the basis lor increased 
fairness and consistency in the scoring of tests. For 
e«.ple. the syste. u.y assign test resolvers to test iten,s 
based on the s=-e racial classification, meaning that the 
test resolver has the sa..e racial classification as the 

1. ^ thf^ resolver is scoring, 

student or respondent whose test the resoi 

The system may also assign test resolvers to test items 
.ased on a different, forced different, or preferred blend 
of classifications. The system monitors consistency m 
scoring based on the selection criteria and, more 
importantly, can change the selection criteria to ensure 
consistent and fair scoring of test items. 

FIG., 9 is a flow chart showing additional typical 
functions of the collaborative scoring. « steps « and 
,3, the system displays the items to resolvers 1 and 2 for 
scoring. The system may further trac. the average scores- 
of resolvers and not send the same test item to two 
resolvers who have provided average scores within a 
predefined range. This also helps to achieve consistency 

If two scorers each have provided 

in scoring. For example, it two scu 

high average scores in the past, as determined by the 
system, these two scorers should preferably not be 
collaboratively scoring the same test items, since it could 

result in "inflated" scores for particular test items. 

The system records the scores from resolvers 1 and 

2 at steps 94 and 95, respectively, and stores such scores 
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ac At Step 97, the system compares 
in a temporary storage 96. At step ' ^^^^^^^ 

.ne scores according to criteria specxfxed xn 

the scores . .^^^^^^^ 

.ppUc...on .epos.o. ^ ^ ^ ^^^^^^^^ 

example, «,u...n, .^at - ^^^^ 
percentage o{ each other. U 

• H ,t step 98, the system records the score 

as determined at step , 

..e .or.-.n-process database "t 3 tep 

— - ToTrr lo vers -st agree. X. the 

step 99 i£ the scores of 

,,rst two resolvers scores do not need to agr . 

ferably transmits the test item to a thxrd 
system preferably tra ^^^^^^ 

, „^ -cure" the discrepancy m the 
resolver to cure resolver 
„ step 100, the system determines .f the 

..ould see the first scores. 

FIG. 10 shows additxonal typical p 
. scoring If the original resolvers 1 and 2 
"rrlTn a or, then the system executes steps 101- 
rte system then typically fi..;^^^^^^^^^^ 

^solver the other resolver-s score at steps 

vho score determined by tne 
, that each resolver can see the score 

At step 103, the system establishes a 
other resolver. At step 

two resolvers. Such a 
co^nnication between the two ^^^^ 

oommnnication lin. may ^ ^ 

Un. so that the scorers can 9 
. the reasoning behind the score provided. «s P 
,5 the reason <i,«rmine a single agreed-upon 

resolvers work together to aet 
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.core for the test ite™. .he syste. prevent the 
«30Xver= 1 an. 2 fro„ receiving another test ite. untrX 
..ey have entered an a,reed-upon score .or the previous 
test Iten,. Finally, at step lOS. the syate. stores the 
.greed-upon score in the .orM-in-process database 

xnstead oi allowin, the resolvers to .or. together 
„ record an agreed-upon score, the system ™ay optionally 
.ecord either a greater value ot the tirst and second test 
scores, a lower value o. the Urst and .econd test scores, 

1 \= the first and second test scores, 
or an average value ot the tir 

II the collaborative scoring criteria specrfxes 

...t the third resolver should arbitrate the 

.„d determine a score, then the syste. displays scores fro. 

, 0 at Step 106 for resolver 3. The 

the resolvers 1 and 2 at step 

1 ^^ then typically enters a score 
third resolver (resolver 3) then typ jr 

for the test ite. at step 107, and the syste» records the 
score in the «or.-in-process database at step 108. 

I£ the collaborative scoring requirement specrfres 
..at the third resolver should not see the first two 
scores, then the syste. executes steps 109-111. At step 
109, the system displays the test item for the third 
^solver. The third resolver then typically enters a score 
at step 110, and the system records the score in the wor.- 
in-process database at step 111. 



5 

r>„aTitY Items_ 



19 



PIG. 11 is . typical flow chart of the u.e o£ 

galley lten,s to cheOc and monitor the accuracy of the 
scoring for selected test resolvers in order to maintain 
consistent and high quality scoring of test ite„s. At step 
112 the system determines or receives the quality 
criteria. The cr^ality criteria -ay be, for example, a 
predetermined test item with a ,cnown "correct" score. 

waits for a scheduled quality 
The system then waits lot 

checK at step 113. « the quality cheC, the system, at 

.esolver. At step 116, the system updates the resolver-s 
quality profile based on the evaluation at step 115. If 
.he resolver should receive a quality result, as determined 
at step 117, the system displays the quality profile to the 

quality profile to a manager for subsequent review. At 
step 120, the system takes action required to assure 
scoring accuracy. 

1 M I lli-in<t--'"T '"-^ Feedback 

FIGURE 12 is a flow chart of typical resolver 
monitoring and feedback. The primary factors in monitoring 
performance typically include. (1) validity, (2) 
; reliability; and (3, speed. In monitoring these factors, 
the system promotes repeatability of scoring. These 
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factors ..y be monitored by tracking a resolver-s 
pertonnance a,ain=t past per.o«a„ce of the resolver or 

against some known goal. 

validity IS typically measured by determining .£ 
particular resolver is applying the scoring key correctly 

expert would score the same Items. Reliability is 
typically measured by determining if a particular w.ll 
resolve the same test item the same way over^ time 
(providing consistent scoring, . Speed" is typlc.lly 
measured by comparing a resolver-s scoring rate with past 
scoring rates the resolver or other scoring rates, such 
as average scoring rates or benchmark scoring rates. 

At step 121, the system typically continually 
monitors the resolver's performance and updates the 
performance. Honitorin, the resolver's performance may 
include, as explained above. «>nitoring the resolver-s 
validity, reliability, and speed in resolving test Items. 
^ system periodically, according to predefined criterra, 
performs performance checks of the test resolvers. 
predefined criteria may include, for example, a time 
period, recalls (how often a resolver evaluates his or her 
^ work,, requesting help, the number of agreements among 
multiple resolvers, the amount of deviation between the 

= »„<i a known score, which may be determined 
resolver 's score and a Known = , 

r.^ i-hf^sG deviations; the 
using quality items; the frequency of these 



speed « which a resolvar enters a response durln, 
resolvm. Of «st the length o, tl»e between scores 

entered a test resolver, a test resolver's previous 

,,erage scorin, rates of other test resolvers, or so^e 

predeter«ined benchmark scoring rate. 

« step 122, the syste» determines whether rt rs 
time for a scheduled performance checK according to the 
predetermined criteria. If it i= time for a performance 

c;teD 123 compares the resolvers' 
check, the system at step i/^ ^ y 

current performance, as determined at step 121, with the 
stored performance criteria. At step 124, the system 
determines if there is a discrepancy in the resolver's 
performance according to the predetermined criteria. For 

. 4-vi£a -r^asoiver's current 

example, the system may determine if the resolver 

scoring rate is within a predefined percentage of the 
average scoring rate in order to ensure efficient scoring 
by the test resolver. If there is no discrepancy, the 
system returns to monitoring the resolver's performance. 
X„ addition, the system may store the resolver- s current 
performance values for later processing. Otherwise, the 
system reports the discrepancy at step 125. 

„ step 126, the system determines if it should 
recommend a breaK in scoring to the resolver. If according 
, ,„ predetermined performance criteria, the system should 
recommend a breaK in scoring, then the system signals the 
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.esolver at step 138 to halt scorin,. Predefined 
performance criteria „ay include, for example, dev.at.ons 
i„ the resolver-s validity, reliability, or .peed of 
resolvin, test items. Examples of predefined performance 

«c;nect to the monitoring 
criteria are provided above wxth respect 

of resolvers' performance. 

«.e„ the resolver stops scoring, the system may 
provide the resolver with the option of rec^^esting 
diversionary activities. Diversionary activities are 
designed to provide the test resolver ."ith a rest period 

•r,rr to increase efficiency. Examples 
and "break" from scoring to increas 

^ .riiivities include computer games and cross 
of diversionary activities 

lo. If the resolver has requested such 
word puzzles, n 

«.tivities, as determined at step 129, then 
diversionary activities, 

the system transmits a diversionary activity to the 
.esolver at step 130. Otherwise, the system returns to 
monitoring the resolver. scoring rate when the resolver 

resumes the scoring. 

If the system at step 126 does not recommend a 
.rea. in scoring .ased on the discrepancy, then the system 

•^^ t-h*. resolver with diversionary 
may optionally provide the resoive 

• A ^i- <5teo 127. If the resolver 
activities as determined at step 1^ 

Should receive the diversionary activities, then the system 
sends such activities to the resolver at step 130. 
, Otherwise the system returns to monitoring the resolver s 

scoring rate. 



nn-Ltne c-^rinq Guide 

FIGURE 13 is a flow chart o£ a typical on-line 
scorm, guide syste-n. The on-xme scoring guide Increases 
scoring efficiency by allowing the resolver to request 
scoring rules In order to assist in scoring a particular 
test item. In response to the re^iest, the system displays 
scoring rules corresponding to a test question for the test 
item currently displayed to the resolver. A resolver may 
thus quioay have specific scoring rules available on-line 
while scoring test Items. This promotes scoring efficiency 
and reduces unnecessary break times resulting from 
determining how to score a particular test Item. 

At step 131, the system sends a test Item to a 
resolver for scoring and displays the test Item at step 
132. H the resolver has requested scoring rules, as 
determined at step 133, then the system Interrogates a 
stored scoring guide to locate scoring rules that 
correspond to a test question for the test item currently 
displayed to the resolver. The system retrieves those 
particular scoring rules at step 135 and displays them to 
the resolver at step 136. The system preferably uses a 
multi-tasking environment In order to simultaneously 
display the scoring rules and the test Item. At step 134, 
the system waits for the resolver to score the test item. 
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At step 137, the system stores the test score entered by 
the resolver into the work-in-process storage. 

AS described above, the present invention is a 
system that processes test items. The various functions 
used in processing the test items promote efficient, high 
quality, and consistent scoring of test items. 

While the present invention has been described in 
connection with the preferred embodiment thereof, it will 
be understood that many modifications will be readily 
apparent to those skilled in the art, and this application 
is intended to cover any adaptations or variations thereof. 
For example, a different hardware configuration may be used 
without departing from the scope of the invention and many 
variations of the processes described may be used. It is 
manifestly intended that this invention be limited only by 
the claims and equivalents thereof. 
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